feat: Add support for List data types #39

mgrazianoc · 2025-06-16T16:04:36Z

Rationale within the changes

This PR refactors and extends support for nested types in the Arrow integration. The current implementation of ArrowNestedType is tailored primarily for data structs, as seen in StructBufferBuilder. However, it lacks broader support and certain expected functionalities, such as loadStructArrayBuilder.

To address this, the following improvements have been made:

Renamed ArrowNestedType to ArrowTypeStruct to align with naming conventions used elsewhere in the codebase.
Introduced initial support for ArrowTypeList, including nested lists.

For simplicity, instead of introducing a dedicated subtype for lists, this PR uses an interface of [Any?]?. If this approach proves insufficient, there are more explicit alternatives that can be explored.

NOTE: Work on ArrowCExporter and ArrowCImporter has been intentionally deferred. These components require a deeper understanding of memory ownership and child parsing, and I believe it's better to be addressed in a future PR, unless it's strict necessary.

What's Changed

Renamed ArrowNestedType -> ArrowTypeStruct.
Added support for ArrowTypeList, including nested lists.
Implemented ListArray with basic .asString formatting.
Added ListArrayBuilder.
Extended ArrowArrayBuilder to support the .list type.
Implemented loadStructArrayBuilder and loadListArrayBuilder.
Introduced ListBufferBuilder.
Added ArrowReader.loadListData.
Added makeListHolder.

Are these changes tested?

Tests are included in ArrayTests.swift. It's also working on internal applications, including integration with ArrowFlight.

Closes #16.

Copilot

Pull Request Overview

This PR extends the Arrow integration to support List data types while refactoring nested type handling. Key changes include renaming ArrowNestedType to ArrowTypeStruct, adding a new ArrowTypeList and corresponding builders/arrays, and updating relevant tests to cover both primitive and nested lists.

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
Arrow/Tests/ArrowTests/TableTests.swift	Updated closure signature and return type for struct builders
Arrow/Tests/ArrowTests/IPCTests.swift	Updated schema construction to use the new ArrowTypeStruct
Arrow/Tests/ArrowTests/ArrayTests.swift	Added tests for ListArray functionality with both primitive and nested lists
Arrow/Sources/Arrow/ProtoUtil.swift	Refactored type creation for struct and added support for list fields
Arrow/Sources/Arrow/ArrowWriter.swift	Updated list handling by replacing ArrowNestedType with ArrowTypeStruct
Arrow/Sources/Arrow/ArrowType.swift	Renamed ArrowNestedType → ArrowTypeStruct and added ArrowTypeList
Arrow/Sources/Arrow/ArrowReaderHelper.swift & ArrowReader.swift	Added list data loading functions
Arrow/Sources/Arrow/ArrowBufferBuilder.swift	Replaced nested type usage and added ListBufferBuilder
Arrow/Sources/Arrow/ArrowArrayBuilder.swift	Introduced ListArrayBuilder and updated builder resolution
Arrow/Sources/Arrow/ArrowArray.swift	Added ListArray implementation with subscript and asString

Arrow/Tests/ArrowTests/ArrayTests.swift

kou · 2025-06-18T08:14:28Z

@abandy You may want to review this.

abandy · 2025-06-18T12:06:36Z

@abandy You may want to review this.

Will do, thanks!

Arrow/Sources/Arrow/ArrowArrayBuilder.swift

Arrow/Sources/Arrow/ArrowBufferBuilder.swift

Arrow/Tests/ArrowTests/ArrayTests.swift

Arrow/Sources/Arrow/ArrowBufferBuilder.swift

Arrow/Sources/Arrow/ArrowArray.swift

… functionality

Co-authored-by: Sutou Kouhei <[email protected]>

Co-authored-by: Copilot <[email protected]>

…entations

mgrazianoc · 2025-08-14T17:37:31Z

@abandy and @kou, sorry for the delay, priorities moved to other places. I think I've managed to resolve the issues, considering what could be more appropriate for the implementation details and interface of the code.

Any questions, please let me know.

abandy · 2025-08-17T16:45:51Z

@abandy and @kou, sorry for the delay, priorities moved to other places. I think I've managed to resolve the issues, considering what could be more appropriate for the implementation details and interface of the code.

Any questions, please let me know.

@mgrazianoc Initial review it lgtm! I will try to find sometime later this week to go over again.

mgrazianoc · 2025-09-10T23:02:09Z

Hi @abandy , do you think we could have this merged by the next weeks? Anything I can help, please let me know.

willtemperley · 2025-10-22T09:42:51Z

@kou maybe we could consider merging this before migrating to the standard project structure? Otherwise this will need rebasing and the authors have been patiently waiting to get it merged for more than one month.

kou · 2025-10-22T11:40:30Z

OK. Could you review this before we merge this?

willtemperley

In general it looks good, but I think there should be a way to specify the nullability of primitives in the list array.

Maybe an API could be added to create a default list field type, similar to arrow-rs. e.g.:

extension ArrowField {
  public init(listFieldWith type: ArrowType, isNullable: Bool) {
    self.init(
      name: "item",
      type: type,
      isNullable: isNullable
    )
  }
}

Arrow/Sources/Arrow/ArrowType.swift

…mplementations

willtemperley

The API for ArrowTypeList now allows element nullability, but this isn't being passed to the ListArrayBuilder. Even if non-nullability isn't actually enforced (a separate issue) I think the public API should allow this to be specified.

Arrow/Sources/Arrow/ArrowArrayBuilder.swift

Arrow/Sources/Arrow/ArrowType.swift

willtemperley

I think the ArrowTypeList needs to be passed to the ListArrayBuilder to specify element nulllability.

…related tests

mgrazianoc · 2025-10-30T18:20:32Z

@willtemperley let me know if the new changes conforms to your concerns and rationale. Thanks for being active on feedbacks

willtemperley

+1

kou · 2025-11-01T21:34:10Z

Thanks both! I'll merge this.

mgrazianoc force-pushed the mgraziano/list-support branch from 28f225d to 0253f70 Compare June 17, 2025 17:35

kou requested a review from Copilot June 18, 2025 08:12

Copilot AI reviewed Jun 18, 2025

View reviewed changes

Arrow/Tests/ArrowTests/ArrayTests.swift Outdated Show resolved Hide resolved

kou changed the title ~~GH-16: Add support for List data types~~ feat: Add support for List data types Jun 18, 2025

kou reviewed Jun 19, 2025

View reviewed changes

abandy reviewed Jun 19, 2025

View reviewed changes

Arrow/Sources/Arrow/ArrowBufferBuilder.swift Show resolved Hide resolved

Arrow/Sources/Arrow/ArrowArray.swift Outdated Show resolved Hide resolved

mgrazianoc marked this pull request as draft August 14, 2025 15:13

Marco added 2 commits August 14, 2025 12:15

fea(Arrow)t: Implement ListArray and ListArrayBuilder with associated…

1d3b817

… functionality

refactor: Rename ArrowNestedType to ArrowTypeStruct for clarity

c4f7ef0

mgrazianoc force-pushed the mgraziano/list-support branch 2 times, most recently from 0f11175 to c4f7ef0 Compare August 14, 2025 16:07

mgrazianoc and others added 5 commits August 14, 2025 13:50

Apply suggestion from @kou

93ce758

Co-authored-by: Sutou Kouhei <[email protected]>

Update Arrow/Sources/Arrow/ArrowBufferBuilder.swift

115751b

Co-authored-by: Sutou Kouhei <[email protected]>

Update Arrow/Sources/Arrow/ArrowArrayBuilder.swift

adfd96b

Co-authored-by: Sutou Kouhei <[email protected]>

Update Arrow/Tests/ArrowTests/ArrayTests.swift

4fcbf2d

Co-authored-by: Copilot <[email protected]>

refactor: Rename StructArray to NestedArray and update related implem…

b8418c1

…entations

mgrazianoc marked this pull request as ready for review August 14, 2025 17:35

swiftlint --fix

5063daa

willtemperley suggested changes Oct 28, 2025

View reviewed changes

Arrow/Sources/Arrow/ArrowType.swift Outdated Show resolved Hide resolved

mgrazianoc force-pushed the mgraziano/list-support branch from 59a4f6d to 5063daa Compare October 29, 2025 19:47

refactor: Update ArrowTypeList to use ArrowField and adjust related i…

0671c26

…mplementations

mgrazianoc requested a review from willtemperley October 29, 2025 20:56

willtemperley reviewed Oct 30, 2025

View reviewed changes

Arrow/Sources/Arrow/ArrowArrayBuilder.swift Outdated Show resolved Hide resolved

Arrow/Sources/Arrow/ArrowType.swift Show resolved Hide resolved

willtemperley suggested changes Oct 30, 2025

View reviewed changes

refactor: Update ListArrayBuilder to accept ArrowTypeList and adjust …

cf7776d

…related tests

willtemperley approved these changes Nov 1, 2025

View reviewed changes

kou merged commit d34e65f into apache:main Nov 1, 2025
11 of 15 checks passed

feat: Add support for List data types #39

feat: Add support for List data types #39

Uh oh!

Conversation

mgrazianoc commented Jun 16, 2025 • edited by kou Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale within the changes

What's Changed

Are these changes tested?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

kou commented Jun 18, 2025

Uh oh!

abandy commented Jun 18, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mgrazianoc commented Aug 14, 2025

Uh oh!

abandy commented Aug 17, 2025

Uh oh!

mgrazianoc commented Sep 10, 2025

Uh oh!

willtemperley commented Oct 22, 2025

Uh oh!

kou commented Oct 22, 2025

Uh oh!

willtemperley left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

willtemperley left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

willtemperley left a comment

Choose a reason for hiding this comment

Uh oh!

mgrazianoc commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

willtemperley left a comment

Choose a reason for hiding this comment

Uh oh!

kou commented Nov 1, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mgrazianoc commented Jun 16, 2025 •

edited by kou

Loading

mgrazianoc commented Oct 30, 2025 •

edited

Loading